Repository navigation
Conversation
MultiTurnRLEvaluator copied only six hard-coded hyperparameter names (sampling_temperature, top_p, max_tokens, ...) into the JobConfigDocument. The hub MTRLEvaluation recipes use different names (temperature, sampling_top_p, sampling_max_tokens, rollout_*), so user-set values were silently dropped and jobs always ran with service defaults (e.g. a 4096 output-token cap). Follow MultiTurnRLTrainer: snapshot the hub defaults when the spec loads and send every value the user changed, keyed by the recipe's own names. Values from recipe/overrides are honored, list values such as pass_k_values are forwarded, and a failed hub lookup now logs a warning. Behavior change: values left at their recipe defaults are no longer sent; the service applies the recipe defaults instead. --- X-AI-Prompt: Fix MultiTurnRLEvaluator so hyperparameters such as the output token cap are forwarded under the hub recipe's names, following how MultiTurnRLTrainer sends user-changed values. X-AI-Tool: Kiro CLI
Shorten docstrings and inline comments added in the previous commit, and drop the hard-coded parameter list from the hyperparameters docstring since the names come from the hub recipe. --- X-AI-Prompt: Reduce the over-explanatory comments added in the MTRL evaluator hyperparameter fix and keep only the useful ones. X-AI-Tool: Kiro CLI
There was a problem hiding this comment.
🤖 AI code review
This PR fixes MultiTurnRLEvaluator silently dropping user-set evaluation hyperparameters by keying them under the recipe's own parameter names instead of a hard-coded list, sending only values that differ from the hub defaults, and upgrading the hub-fetch-failure log to WARNING. The approach mirrors MultiTurnRLTrainer and is backed by new unit tests. The core logic is sound; one real gap remains around list-valued hyperparameters supplied via recipe/overrides.
Reviewed commit f17827d. Automated review; verify before acting.
| Returns: | ||
| Dict[str, str]: Changed values, keyed by recipe parameter name. | ||
| """ | ||
| final = self.hyperparameters.to_dict() |
There was a problem hiding this comment.
🟠 Medium: List-valued hyperparameters (e.g. pass_k_values) set via recipe or overrides are silently dropped. _get_effective_hyperparameters() uses flatten_resolved_recipe, which explicitly skips list values (elif not isinstance(v, list) in recipe_resolver.flatten_resolved_recipe), so the loop here never sees them. The only list values that survive are those captured from self.hyperparameters.to_dict(), i.e. those set via direct attribute assignment. The PR body states "This covers evaluator.hyperparameters.<name>, recipe and overrides" and "List values such as pass_k_values are forwarded", but the recipe/overrides path for list values does not work. The new test_list_values_are_forwarded only exercises direct assignment and test_overrides_dict_is_forwarded only uses scalars, so this gap is untested. Consider merging list values from the resolved recipe before dropping them, or documenting the limitation.
…overrides flatten_resolved_recipe skips list values, so pass_k_values set through recipe or overrides was dropped. Take list values from the resolved recipe's inference section as well. --- X-AI-Prompt: Address the review comment that list-valued hyperparameters like pass_k_values set via recipe or overrides are silently dropped, without changing which keys are forwarded. X-AI-Tool: Kiro CLI
There was a problem hiding this comment.
🤖 AI code review
This PR fixes MultiTurnRLEvaluator silently dropping user-set evaluation hyperparameters. The old _build_job_config_doc copied six hard-coded names (eval_group_size, sampling_temperature, top_p, max_tokens, pass_k_values, success_threshold) that don't match the hub MTRLEvaluation recipe names (temperature, sampling_top_p, sampling_max_tokens, rollout_*), so hparams.get() returned None and the values were discarded. The new _get_user_changed_hyperparameters captures hub defaults and forwards only user-changed values under the recipe's own names, closely mirroring MultiTurnRLTrainer._build_training_config/_apply_recipe_to_hyperparameters. The change is consistent with the established trainer pattern, well covered by 7 new unit tests, and I found no blocking issues.
Reviewed commit 69093b0. Automated review; verify before acting.
| ) | ||
| for k, v in values.items(): | ||
| if v is not None: | ||
| final[k] = v if isinstance(v, str) else str(v) |
There was a problem hiding this comment.
🟡 Low: In _get_user_changed_hyperparameters, recipe/override-sourced values are stringified (final[k] = str(v)) before the final comparison v != defaults.get(k), but defaults holds native-typed values from to_dict() (e.g. int 4096). So a value supplied via recipe/overrides that is equal to the recipe default (e.g. sampling_max_tokens: 4096) still compares unequal ("4096" != 4096) and gets sent in HyperParameters, whereas the same value set via direct attribute assignment is correctly suppressed. The effect is harmless since the service applies the same default, and MultiTurnRLTrainer has the identical behavior, so this is only a minor inconsistency. The new test test_value_equal_to_default_is_not_sent only exercises the direct-assignment path and wouldn't catch this; consider a test for the recipe/overrides-equals-default case if you want the suppression to be exact.
Issue
MultiTurnRLEvaluatorsilently ignored most user-set evaluation hyperparameters.For example,
sampling_max_tokenscould not be raised above the 4096 default.Root cause
_build_job_config_doccopied six hard-coded names intoEvaluationConfig.HyperParameters:eval_group_size,sampling_temperature,top_p,max_tokens,pass_k_valuesandsuccess_threshold. The hubMTRLEvaluation recipes use
temperature,sampling_top_p,sampling_max_tokensand
rollout_*instead.hparams.get()returnedNonefor each mismatched name,and the value was dropped without a warning. I confirmed the names for all four
MTRL models in SageMakerPublicHub: gemma-4-31b-it, qwen3-6-27b, nova-lite-v2 and
gpt-oss-20b.
Fix
This follows
MultiTurnRLTrainer._build_training_config:evaluator.hyperparameters.<name>,recipeandoverrides.pass_k_valuesare forwarded.Before this change,
eval_group_size,pass_k_valuesandsuccess_thresholdwerealways sent, even when they were at their defaults. Now
HyperParameterscontainsonly values that differ from the recipe defaults, and the block is omitted when
nothing changed. The service applies the recipe defaults in that case, so the
values used are the same, but the generated pipeline JSON is smaller. If the
service's defaults ever differ from the hub spec's defaults, values the user didn't
set will follow the service. This matches how
MultiTurnRLTraineralready behaves.Testing
test_mtrl_evaluator.py, using a mocked copy of the hubspec. 5 of them fail on the old code.
tests/unit/train/evaluate/andtest_trainer_recipe_integration.py:639 passed, 7 skipped.
sampling_max_tokens=8192,rollout_timeoutandpass_k_valuesappear in theJobConfigDocument, and 16384 is rejected because it exceeds the max of 8192.
blackandflake8report no problems.